Terapixel Image Processing and Simulation with Distributed Halide

Many image processing and simulation tasks are naturally expressed as a pipeline of small computational kernels known as stencils. Halide is a popular domainspecific language and compiler designed to implement stencil algorithms. Halide uses simple language constructs to express what to compute and a separate scheduling co-language for expressing how to perform the computation. This approach has demonstrated performance comparable to or better than hand-optimized code. Until now, Halide has been restricted to parallel shared memory execution, limiting its performance and applicability to tomorrow’s terapixel image processing tasks. In this thesis we present an extension to Halide to support distributed-memory parallel execution of stencil pipelines. These extensions compose with the existing scheduling constructs in Halide, allowing expression of complex computation and communication strategies. Existing Halide applications can be distributed with minimal changes, allowing programmers to explore the tradeoff between recomputation and communication with little effort. Approximately 10 new of lines code are needed even for a 200 line, 99 stage application. On nine image processing benchmarks, my extensions give up to a 1.4× speedup on the same number of cores over regular multithreaded execution by mitigating the effects of non-uniform memory access. The image processing benchmarks achieve up to 18× speedup on a 16 node testing machine and up to 57× speedup on 64 nodes of the NERSC Cori supercomputer. A 3D heat finite-difference simulation benchmark achieves linear scaling from 64 to 512 Cori nodes on a 10, 0003, or 1 terapixel, input. We also demonstrate scalability results for two of the image processing benchmarks on 1 terapixel inputs, and make the argument that supporting such large scale is essential for tomorrow’s image processing and simulation needs. Thesis Supervisor: Saman Amarasinghe Title: Professor of Electrical Engineering and Computer Science